Method and apparatus for communicating data between two hosts

ABSTRACT

A method for communicating video data between a first host and a second host. The method may comprise encoding a video signal producing an encoded video signal at the first host. The method may also comprise transmitting, by the first host, the encoded video signal to the second host. The encoding and transmitting may both be performed by a first single multimedia thread of execution associated with an operating system at the first host. Other embodiments are also disclosed.

RELATED APPLICATIONS

This application claims priority under 35 U.S.C. § 119(e) to U.S.Provisional Application Ser. No. 60/512,667 entitled “Method andApparatus for Communicating Video Information,” filed on Oct. 20, 2003,which is herein incorporated by reference in its entirety.

BACKGROUND

The invention relates generally to the transmission of real-time audiodata, especially and/or video information, and more particularly tosystems and methods for video conferencing.

There are many systems and techniques for transmitting video informationespecially real-time video information combined with corresponding audioor other information such as document displays. Some effectiveconventional techniques involve special transmission and receptionsystems and require dedicated communication links to encode, transmit,receive and decode video or other information. The encoding,transmission, and decoding operations are generally resource intensivein terms of the processing (e.g., memory, CPU speed) and transmissionrequirements (e.g., communication link bandwidth) necessary to providean adequate video presentation. However, such special systems aregenerally expensive to own and operate and therefore are not availableto an average consumer.

Many commercial products including hardware and/or software componentshave become available to the average consumer for transmitting videoinformation over public networks, such as the Internet. These systemsmay be, for example, coupled with a personal computer for use over theInternet or other communication network. For example, a videoconferencing or video distribution system may be configured to transmitvideo information over the Internet among a group of PCs. However, dueto the substantial resource requirements necessary for transmitting suchinformation, and the limited and/or unreliable resources available onpublic networks, performance of such systems generally fall short ofexpectations, and such systems are rendered less-usable than moreexpensive specialized systems.

The quality of a combined audio-video communication as perceived by auser is highly correlated to both the overall latency of thecommunication and to the difference between the latency of the audiocommunication and the latency of the video communication between thesending and receiving systems.

For example, latency is commonly experienced during a typical cell phonecall, in that during the conversation, there are delay periods due tolatency that are periodically perceived by the user. In such a scenario,the user feels like the call is not in real time, and this latencyaffects the actual and perceived quality of the cell phone connection.With respect to video transmission, users experience latency as time lagor “jumpiness” between consecutive video frames. Such performance ispresent, for example, in conventional software video conferencingsolutions.

Video quality may particularly suffer when transmitting data overcommunication networks such as the Internet. Due to bandwidthavailability and latency, conventional video conferencing solutionsgenerally provide frame rates between 2 and 10 frames per second. Forinstance, the system CUseeMe (available from First VirtualCommunications, Redwood City, Calif.) allows video transmission betweenhosts using a video conference server provides a 2-3 frame per secondvideo signal with a latency of 450 milliseconds between capture at onehost and presentation at another host. Another Internet teleconferencingsystem, NetMeeting (available from the Microsoft Corporation, Redmond,Wash.) delivers video information at approximately 6-10 frames persecond with a latency of approximately 230 milliseconds. Further, mostof these conventional systems are unable to deliver an acceptable imagesize at a good quality. Conventional systems that operate over publicnetworks such as the Internet are capable of delivering, at most,320×240 pixel video data at 6-10 fps. By contrast, video delivered atapproximately 24-30 frames per second (theatrical motion pictures areshown at 24 fps, while television displays at 30 fps) provides aperception to the viewer of full-motion video. Therefore, it would bebeneficial to have a system that delivers a quality video signal withoutlatency or jumpiness between successive frames.

Contributions to latency may include and are not limited to intermediatesystems, network latency and latency due to processing. Someconventional systems use intermediate systems to handle video datatransmitted between hosts. More specifically, video data from one hostis transmitted to another host through an intermediate system.Intermediate systems may include various security and network componentsas well, including but not limited to firewalls, routers and others. Theextra handling performed by these systems, which includes receiving,processing, buffering, and other steps at the intermediate system, addslatency to the transmission.

In some conventional systems, there is significant network latency dueto one or more components of the network connection. This latency isdue, for example, to additional network transmissions throughintermediate systems as discussed above, and latency in creating andestablishing network connections. Further, there are other problems, inaddition to latency, with establishing connections through firewalls andother secure networking systems, as discussed below.

There are other contributing factors to latency at either or both of thereceiving and transmitting hosts. For instance, latency is added ateither the sending or receiving host due to over-handling of the videodata. Excess buffering, thread-to-thread copying of video data, andother factors contribute to this type of latency.

Modern computer networks enable communication between computers in partby assigning each connected computer an address (for example, anInternet Protocol (IP) address), and one or more ports through whichcommunication may proceed between the assigned IP addresses. Once alogical connection between computers is established, various techniquesassure the authenticity of the ongoing information transfers, forexample assigning sequence numbers to packets forming part of an ongoingconnection. Some network interconnectivity features and security systemssuch as firewalls, network address translation (NAT) features, andothers conventionally used, interfere with efficient, latency-free,real-time transmission of high bandwidth information, such as videoinformation. What is needed, therefore, is an improved method forcommunicating video information.

When communication between a first host computer and a second hostcomputer is desired, and the computers are connected to each otherthrough a communications network without an intervening firewall or anintervening device performing NAT, either host may initiate theconnection by simply sending a suitable message addressed to a suitableport for the message type at the address of the other host computer.Communication using a client/server architecture, or a peer-to-peerarchitecture, or any other suitable architecture can be initiated inthis way, absent an intervening firewall or NAT device.

Communication without an intervening firewall or NAT device sometimesoccurs when host computers are connected to a common local area network(LAN), such as a corporate network or home network. More commonly, whenthe host computers are interconnected through a wide area network (WAN),such as the Internet, and sometimes in LAN configuration, one or bothhost computers may be connected to the WAN through a firewall or NATsystem. A firewall or NAT system has the effect of partially orcompletely masking, from computer systems reachable through the WAN, thehost computers behind such firewall or NAT systems. This masking isperformed by rejecting unexpected messages, i.e., messages sent toclosed or incorrect addresses or ports or messages purporting to be partof an ongoing exchange, but having incorrect sequence numbers.

To make a conventional feature such as worldwide web browsing work,several operations occur. Connections from a local host computer to aremote server are initiated by the local computer. If the local computeris behind a firewall, the local computer initiates communication with adesired server, through the firewall. By initiating the communicationthrough the firewall, the local computer instructs the firewall to allowcertain types of communications back from the server for a certainperiod of time. When the server replies using a correct address, portand sequence number, the firewall recognizes the response as expectedand passes the response on to the local computer. While the server mayalso be located behind a firewall, that firewall conventionally has aknown port open to inbound traffic, so that certain types of contactwith the server are permitted by the firewall.

To allow a direct peer-to-peer connection, a port is conventionallyopened in the firewall of each participant that is connected to thenetwork through a firewall, so that contact with any participant may beinitiated by any other participant, however, this leaves eachparticipant with at least one port of their firewall open and vulnerableto security breaches. In such applications as video and/orteleconferencing through the Internet, where direct, peer-to-peerconnections are desirable for the purpose of minimizing latency, it ishighly desirable to minimize the security risk to the computers ofparticipants, while also allowing any participant to initiate aconnection with minimal effort.

In some conventional systems, an event loop is used to process videoinformation. This is generally in the form of a single thread (e.g., athread of execution executed by a processor) that executes in aninfinite loop. The thread waits until an event happens, and when anevent occurs, the thread acts upon the event. Generally, only a singleevent can be processed at a time. Other threads can add events to thethread's workload, but the other threads cannot actually handle theseevents. This event loading causes the thread to become overloaded, andtherefore, a particular event (e.g., a video transmission, encoding ordecoding event) that needs to be processed is delayed. A simple exampledescribing this issue is a worker scenario where a particular worker hasmultiple bosses, each of which generates work for the worker to perform.In performing a particular work task for one of the bosses, tasksrequested of other bosses become delayed and must wait for the taskcurrently being performed to be completed.

SUMMARY OF INVENTION

According to one aspect of the invention, methods and apparatuses areprovided that establish a low-latency connection between two hosts. Inone embodiment of the present invention, such a connection may be madeover a public network such as the Internet. However, it should berealized that various aspects of the present invention may be used withany network type (e.g., within an enterprise network, virtual privatenetwork (VPN), etc.) or any combination of networks and networkcommunication types (e.g., Ethernet), and the invention is not limitedto any particular network or combination of networks. Moreover, althoughthe invention will be explained in the context of video conferencing,aspects of the invention have wider data communication applicability.

The human threshold for perceiving latency is approximately 100 msec.Therefore, by communicating a video signal at or below this thresholdbetween two hosts increases the perceived quality of the transmission toa user viewing the video signal.

Firewalls pose a particular latency problem for transmitting data byrequiring reconnection of connections that have timed out. As previouslyexplained, firewalls allow and maintain connections that are initiatedfrom a secure side of the firewall. As also previously explained,reverse traffic from other hosts are allowed in if they can beattributed to an allowed connection initiated from the secure side. Forsecurity reasons, firewalls delete allowed connections after a period ofnonuse.

In one aspect of the invention, a host operating on a secure side of afirewall is adapted to periodically transmit a packet to a directoryserver to maintain that port open to the firewall for receipt of packetsfrom one or more other hosts.

According to one aspect of the invention, it is realized that the lowesttimeout value on commercially-available firewalls is 30 seconds.Therefore, periodic keep-alive messages are transmitted by the host tothe directory server with a period shorter than the timeout to maintainthe connection established through the firewall.

Port connections to a particular host may be registered with thedirectory server, and port connection information for the particularhost can be communicated to another host for the purpose of establishinga connection between the two hosts. That is, video traffic is sent bythe other host to the first host, located to the registered port at thefirewall. Information stored in the directory server may include an IPaddress, a port number, and other security information expected by thefirewall. In this manner, the first host on the secure side of thefirewall maintains a connection through which another host can passvideo traffic. It is conversely true that the second host may also belocated beyond a firewall, and may also register its port informationwith the directory server in order to facilitate the transfer of videoinformation from the first host to the second host through the firewallassociated with the second host. In this manner, a true host to host (orpeer to peer) connection may be established between hosts. Because theconnection does not involve an intermediate server in the ordinarytransmission of data packets (e.g., video data, audio data, etc.)latency is reduced.

A method for communicating data between at least a first host and asecond host comprises: identifying, at a server, address information ofthe first host, to which the second host may communicate data through anetwork security system coupled between the first host and the server;and communicating, from the second host to the first host, data usingthe address information of the first host. The method may furthercomprise: identifying, at the server, address information of the secondhost, to which the first host may communicate data, through a networksecurity system coupled between the second host and the server; andcommunicating, from the first host to the second host, data using theaddress information of the second host. The second host may be adaptedto perform the act of communicating without use of an intermediateserver. The method may also further comprise communicating periodically,from the first host to the server, through the network security system,so as to maintain an open communication channel through the networksecurity system to the first host at the address of the first host. Theaddress information of the first host may include an address and a portinformation.

Preferably, both video and audio information is transmitted in real ornear-real time between two hosts such that a user of at least one of thehosts perceives a real-time communication, that is, latency should bereduced below 100 mS for each. According to various aspects of thepresent invention, one or more of the above methods may be performed toreduce latency between hosts, and the invention is not limited to anyparticular combination of methods. For instance, separate receive andtransmit threads may be used independently of any network considerationsand still provide a performance improvement in transmitting andprocessing a video signal.

According to another aspect of the present invention, a system isprovided that delivers full-motion video with a low latency to provide anear real-time perception by a user. In one aspect of the presentinvention, video is provided at near telephone (POTS) latency, andtherefore allows the transmission of video information over a publicnetwork such as the Internet and transmission of audio information overa standard telephone connection. It is realized that latency of atelephone signal is approximately 60 msec. Therefore, by communicatingvideo information between hosts near this latency, there is no perceiveddelay between the video and audio information, and therefore, thetransmission of video may be easily added to an existing telephoneconnection. Flexibility is increased, as the telephone is morefrequently used to perform communications (versus the time and effort toestablish a video conference) and therefore, augmenting a standardtelephone call with video is more likely to be performed.

BRIEF DESCRIPTION OF DRAWINGS

The accompanying drawings, are not intended to be drawn to scale. In thedrawings, each identical or nearly identical component that isillustrated in various figures is represented by a like numeral. Forpurposes of clarity, not every component may be labeled in everydrawing. In the drawings:

FIG. 1 is a schematic block diagram of a computer system in whichaspects of the invention may be embodied;

FIG. 2 is a schematic block diagram of the storage system of thecomputer system of FIG. 1;

FIG. 3 is a schematic block diagram of a network topology in whichaspects of embodiments of the invention may be realized;

FIG. 4 is a flow diagram illustrating a method embodying aspects ofembodiments of the invention;

FIG. 5 is a flow diagram detailing the registration process of themethod of FIG. 4;

FIG. 6 is a flow diagram detailing the conference initiation process ofthe method of FIG. 4; and

FIG. 7 is a threading diagram showing process flow and data flowaccording to aspects of an embodiment of the invention.

DETAILED DESCRIPTION

This invention is not limited in its application to the details ofconstruction and the arrangement of components set forth in thefollowing description or illustrated in the drawings. The invention iscapable of other embodiments and of being practiced or of being carriedout in various ways. Also, the phraseology and terminology used hereinis for the purpose of description and should not be regarded aslimiting. The use of “including,” “comprising,” or “having,”“containing”, “involving”, and variations thereof herein, is meant toencompass the items listed thereafter and equivalents thereof as well asadditional items.

Embodiments of aspects of the present invention form and maintain anopen channel for communication between two or more systems (e.g.,computer systems) coupled by a communication network. In one embodiment,one or more intermediate security devices (e.g., firewalls, NAT devices)are interposed between the two or more systems. The open channel may beformed first by communicating with a directory service, and thenestablishing direct communication in which the directory service remainsoutside of the communication path of the channel.

In conventional media transmission systems such as video conferencingand other systems, information is transmitted between computers by wayof an intermediate server. This intermediate server receives informationfrom one computer, buffers the information, and transmits theinformation to one or more other computers. According to one aspect ofthe present invention, a communication method is provided that allowspeer-to-peer communication without an intermediate server system. Adirect peer-to-peer communication reduces the amount of latency incommunicating information (e.g., video, audio) information between asending and receiving system, as the need for buffering information atanother system is eliminated, and multiple receive and transmits of theinformation (e.g., to/from the intermediate server) is no longernecessary. In one aspect of the present invention, it is realized thatlatency in communicating information between systems affects theperceived quality of the communication between systems (e.g., in avideoconferencing system), and it may be beneficial to minimize theamount of latency in communicating the information between systems. Inone embodiment, the directory service system maintains informationrelating to the two or more systems that allows any of the individualsystems to establish one or more direct communication channels withanother system that is registered with the directory service system.

According to another aspect of the invention, messages are periodicallysent to maintain the open connection through an intermediate securitysystem (e.g., NAT device, firewall) such that a system located beyondthe intermediate security system may receive peer-to-peer connectionrequests directly. By maintaining the connection through the securitysystem, latency associated with creating a connection is reduced.

Other aspects of the present invention relate to reducing latency of thecommunication between systems by modifying ways by which the sending andreceiving systems process data. According to one aspect, it is realizedthat existing systems use inefficient programming methods for sending,receiving and processing data.

According to aspects of embodiments of the invention shown in FIG. 7,implementing dual, single threads, one of which, block 701, handlesreceipt of data and one of which, block 702, handles transmission ofdata allows more efficient handling of video data. Receiving andprocessing of a packet, block 701, is made independent of sending videodata, block 702, and receiving and sending can occur concurrentlybecause the threads, blocks 701 and 702, execute independently. In someconventional systems, an event loop is used to process videoinformation. The event loop takes the form of a single thread thatexecutes in an infinite loop. The thread waits until an event occurs,and then the thread acts upon the event. In such a design, only a singleevent can be processed at a time. Other threads can add events to theevent loop thread's workload, but the other threads cannot actuallyhandle these events. This event loading causes the single event loopthread to become overloaded, and therefore, a particular event, e.g., avideo transmission, encoding or decoding event that needs to beprocessed is delayed.

In one embodiment, the system is dual-threaded, having a separatereceiving thread, block 701, and transmission thread, block 702, howeverthe receiving thread is single-threaded as to all receivingfunctionality and the transmission thread is single-threaded as to alltransmission functionality. The sending or transmission thread, of oneembodiment encodes video data and the audio data and places them on thetransmitter. In another embodiment, the sending thread may be a videomultimedia thread that performs encoding and transmission. This threadmay be, for example, a near real-time thread included in the OS (e.g.,Windows NT or other operating system) that calls a function periodicallycorresponding to the encoding function. For example, in Windows, anencoding function may be provided to the OS by a callback at 30 fps. Thefunction is then called every 1/30 second, and this single thread isused to perform the encoding and transmission of data. Because there areno thread-to-thread copies of data, latency is reduced. Also, becausethe thread is at a near-real time priority, the thread experiences lessdelay than a thread having lesser priority.

Similarly, in one embodiment the receiving thread, which is separatefrom the transmitting thread, receives the video data and the audio dataand decodes them for display and/or playback. Decoding may be performedby an operating system callback, as well.

According to embodiments of other aspects of the invention, a dual,single-thread software design reduces processing latency, so thatconventional, low-latency audio signal paths can be used to transmit andreceive an audio signal while a corresponding video signal issimultaneously transmitted and received through a computer communicationnetwork having sufficiently low latency that the video signal and theaudio signal are perceived as being synchronized.

Embodiments of aspects of the present invention may be practiced onspecial purpose or general purpose computers, as now described.

Various embodiments according to the invention may be implemented on oneor more computer systems. These computer systems may be, for example,general-purpose computers such as those based on Intel PENTIUM-typeprocessor, Motorola PowerPC, Sun UltraSPARC, Hewlett-Packard PA-RISCprocessors, or any other type of processor. It should be appreciatedthat one or more of any type computer system may be used to collectinformation and communicate over a network, according to variousembodiments of the invention. Further, the exemplary video conferencingsystem may be located on a single computer or may be distributed among aplurality of computers attached by a communications network.

A general-purpose computer system according to one embodiment of theinvention is configured to perform any of the described videoconferencing functions including but not limited to collecting video oraudio information, transmitting the video or audio information,receiving the video or audio information or performing directoryservices. It should be appreciated that the system may perform otherfunctions, including network communication, and the invention is notlimited to having any particular function or set of functions.

For example, various aspects of the invention may be implemented asspecialized software executing in a general-purpose computer system 100such as that shown in FIG. 1. The computer system 100 may include aprocessor 103 connected to one or more memory devices 104, such as adisk drive, memory, or other device for storing data. Memory 104 istypically used for storing programs and data during operation of thecomputer system 100. Components of computer system 100 may be coupled byan interconnection mechanism 105, which may include one or more busses(e.g., between components that are integrated within a same machine)and/or a network (e.g., between components that reside on separatediscrete machines). The interconnection mechanism 105 enablescommunications (e.g., data, instructions) to be exchanged between systemcomponents of system 100.

Computer system 100 also includes one or more input devices 102, forexample, a keyboard, mouse, trackball, microphone, touch screen, and oneor more output devices 101, for example, a printing device, displayscreen, speaker. In addition, computer system 100 may contain one ormore interfaces (not shown) that connect computer system 100 to acommunication network (in addition or as an alternative to theinterconnection mechanism 105.

The storage system 106, shown in greater detail in FIG. 2, typicallyincludes a computer readable and writeable nonvolatile recording medium201 in which signals are stored that define a program to be executed bythe processor or information stored on or in the medium 201 to beprocessed by the program. The medium may, for example, be a disk orflash memory. Typically, in operation, the processor causes data to beread from the nonvolatile recording medium 201 into another memory 202that allows for faster access to the information by the processor thandoes the medium 201. This memory 202 is typically a volatile, randomaccess memory such as a dynamic random access memory (DRAM) or staticmemory (SRAM). It may be located in storage system 106, as shown, or inmemory system 104, not shown. The processor 103 generally manipulatesthe data within the integrated circuit memory 104, 202 and then copiesthe data to the medium 201 after processing is completed. A variety ofmechanisms are known for managing data movement between the medium 201and the integrated circuit memory element 104, 202, and the invention isnot limited thereto. The invention is not limited to a particular memorysystem 104 or storage system 106.

The computer system may include specially-programmed, special-purposehardware, for example, an application-specific integrated circuit(ASIC). Aspects of the invention may be implemented in software,hardware or firmware, or any combination thereof. Further, such methods,acts, systems, system elements and components thereof may be implementedas part of the computer system described above or as an independentcomponent.

Although computer system 100 is shown by way of example as one type ofcomputer system upon which various aspects of the invention may bepracticed, it should be appreciated that aspects of the invention arenot limited to being implemented on the computer system as shown inFIG. 1. Various aspects of the invention may be practiced on one or morecomputers having a different architecture or components that that shownin FIG. 1.

Computer system 100 may be a general-purpose computer system that isprogrammable using a high-level computer programming language. Computersystem 100 may be also implemented using specially programmed, specialpurpose hardware. In computer system 100, processor 103 is typically acommercially available processor such as the well-known Pentium classprocessor available from the Intel Corporation. Many other processorsare available. Such a processor usually executes an operating systemwhich may be, for example, the Windows 95, Windows 98, Windows NT,Windows 2000 (Windows ME) or Windows XP operating systems available fromthe Microsoft Corporation, MAC OS System X operating system availablefrom Apple Computer, the Solaris operating system available from SunMicrosystems, or UNIX operating systems available from various sources.Many other operating systems may be used.

The processor and operating system together define a computer platformfor which application programs in high-level programming languages arewritten. It should be understood that the invention is not limited to aparticular computer system platform, processor, operating system, ornetwork. Also, it should be apparent to those skilled in the art thatthe present invention is not limited to a specific programming languageor computer system. Further, it should be appreciated that otherappropriate programming languages and other appropriate computer systemscould also be used.

One or more portions of the computer system may be distributed acrossone or more computer systems coupled to a communications network. Thesecomputer systems also may be general-purpose computer systems. Forexample, various aspects of the invention may be distributed among oneor more computer systems configured to provide a service (e.g., servers)to one or more client computers, or to perform an overall task as partof a distributed system. For example, various aspects of the inventionmay be performed on a client-server or multi-tier system that includescomponents distributed among one or more server systems that performvarious functions according to various embodiments of the invention.These components may be executable, intermediate (e.g., IL) orinterpreted (e.g., Java) code which communicate over a communicationnetwork (e.g., the Internet) using a communication protocol (e.g.,TCP/IP).

It should be appreciated that the invention is not limited to executingon any particular system or group of systems. Also, it should beappreciated that the invention is not limited to any particulardistributed architecture, network, or communication protocol.

Various embodiments of the present invention may be programmed using anobject-oriented programming language, such as SmallTalk, Java, C++, Ada,or C# (C-Sharp). Other object-oriented programming languages may also beused. Alternatively, functional, scripting, and/or logical programminglanguages may be used. Various aspects of the invention may beimplemented in a non-programmed environment (e.g., documents created inHTML, XML or other format that, when viewed in a window of a browserprogram, render aspects of a graphical-user interface (GUI) or performother functions). Various aspects of the invention may be implemented asprogrammed or non-programmed elements, or any combination thereof.

Embodiments of aspects of the present invention are now illustrated withreference to a particular network topology as shown in FIG. 3. Anysuitable variation on this topology may alternatively be employed. Forexample, additional routers, gateways, switches, firewalls, servers,hardware components and others not shown may be present in suitableconfigurations in the network. The illustrative topology includes afirst host computer, 301, connected to a communication network, 302, forexample, the Internet, through a firewall, 303, a second host computer,304, connected to the communication network, 302, through a firewall,305, and a directory server, 306, also connected to the communicationnetwork, 302, through a firewall, 307. The method to be describedillustrates the behavior of the system when one party, a calling party,having access to the first host computer, 301, desires to establishcommunication with another party, a called party, having access to thesecond host computer, 304. The first host computer and its user will becollectively referred to as the “calling party” while the second hostcomputer and its user will be collectively referred to as the “calledparty.” As discussed above, the calling party cannot establish directcommunication with the called party. Unexpected packets of informationsent from the calling party to the called party in the attempt toestablish the communication channel will be rejected by the calledparty's firewall system, e.g., firewall 303 or firewall 305.

According to a method embodying aspects of the invention, illustrated bythe flow diagram of FIG. 4, the directory server maintains open channelsto each party, referred to as secondary channels, until a direct channelis established between a calling party and a called party, then stepsout of the path.

Both the calling party and the called party first register theiravailability with the directory server, 4001. Looking first at thecalled party, in order to register, the called party opens a portthrough the firewall (FIG. 3, 303 or 305) protecting the called party bysending a message to the directory server (FIG. 3, 306), through which asecondary channel is established with the directory server, 401.Although the directory server (FIG. 3, 306) is also protected by afirewall (FIG. 3, 307), that firewall has a known open port throughwhich the message is sent. The called party indicates through thatchannel, indeed through the contents of the message sent, itsavailability for communication.

Importantly, the called party maintains the communication channel withthe directory server in an open and available state, by periodicallysending keep-alive messages to the directory server, 402, which mustecho or acknowledge the keep-alive messages back to the called party,403, so as to keep the port open through the firewall (FIG. 3, 303 or305) protecting the called party.

When the calling party desires to register, the calling party performsregistration steps similar to those described in connection with thecalled party, 4001. Thus, the calling party opens a port through thefirewall (FIG. 3, 303 or 305) protecting the calling party. A secondarychannel is then established through the open port, to the directoryserver (FIG. 3, 306). The calling party then indicates to the directoryserver (FIG. 3, 306) its availability for communication, and its desireto communicate with the called party. Like the called party describedabove, the calling party also sends keep-alive messages to the directoryserver, 402, and these messages are echoed or acknowledged back to thecalling party, 403, so as to keep the firewall protecting the callingparty open for receiving messages. The registration messages and thekeep-alive messages may contain the same information, or they maycontain different, specialized information, as desired. Through thecontent of the registration messages from each calling party and fromeach called party, the directory server acquires information identifyingthe addresses and ports through which the calling party and the calledparty are communicating.

When the calling party indicates a desire to establish directcommunication with the called party, 4002, the directory serverinstructs the calling party and the called party to communicate directlythrough each one's established open channel, 4003. When the directoryserver instructs the calling party and the called party, 4003, theinstructions include the address, port and sequence number information,or any other suitable information required to establish directcommunications. Messages from the calling party to the called party orfrom the called party back to the calling party do not pass through thedirectory server, but rather, the messages travel directly from one hostto the other through one or more communication networks.

The processes just briefly described are now described in greater detailin connection with the flow diagrams of FIGS. 5-7.

According to this illustrative embodiment, all parties to a videoconference register with the directory server. The directory serverawaits contact from each host desiring to register with the directoryserver; each host first logging in using a push script (FIG. 5, 501). Ifthe login parameters (for example, email address and password) arevalid, then the server returns a contact list result page, together witha group of triggered parameters attached to the URL for the contact listresult page (FIG. 5, 502). The triggered parameters include a user IDnumber for the host, a session ID number for security purposes which isvalid until the host logs out, the directory server name, i.e. the DNSname of the server to contact for the secondary channel to beestablished and the directory server port, i.e. the port on thedirectory server to connect to. Other triggered parameters could be useddepending upon implementation, and other implementations may not requireall of the triggered parameters given above. After registration iscomplete, the keep-alive process is commenced (FIG. 5, 503).

The keep-alive process involves sending packets through the secondarychannel between the host and the directory server on a regular basis tokeep the channel open. The packet may contain useful status informationto be processed or displayed by the server, such as status information(e.g., busy or away) as well as the host computer's internal IP address,the host computer's internal port, the host computer's internal hardwareMAC address, as well as any other useful information that may be desiredby the system designer. In this exemplary embodiment, the directoryserver records all the information mentioned above, including thephysical source IP address and the port from which the packet was sent.Because firewall and NAT devices change the source IP address and portas packets pass through them, this information is useful for latersending packets directly back to a particular host. For example, if theinformation collected by the directory server shows a mismatch betweenthe internal IP address and port and the external IP address and portfor a particular packet sent by a particular host, then the host isprotected by or behind either a firewall or NAT device. This informationhelps establish which local area network a particular host may be amember of, as will be required below. The host should send thesekeep-alive packets periodically with a time between packets short enoughthat the firewall system protecting the host computer will be fooledinto thinking that the directory server and the host computer are incontinuous communication, and thus leave the channel open to receiveunsolicited packets at the host when needed. Once at least two parties,i.e. host computers have registered with the directory server, aconnection can be initiated.

To initiate a conference, as shown in FIG. 6, first the calling partyselects a called party from the contact list page displayed uponregistration, at block 601. The selection may be made by any suitablemeans, for example by using a pointing device to click on, and thusselect, the called party's nickname in a displayed list of potentialcontacts. Several initial verifications are performed, for example,billing verifications and a determination as to whether the callingparty and the called party are on the same local area network ordifferent local area networks, at block 602. A conference ID is computedat block 603, so that further communication in connection with thepresent conference can be identified as such. If the calling party andthe called party are on different networks, the discovery process 600may commence.

An example discovery process 600, according to one embodiment, proceedsas follows. The discovery process, 600, leads to an establishment ofdirect communications.

First the directory server sends discovery start packets, at blocks 604and 605, to the calling party and to the called party. The discoverystart packets can be sent in parallel, as shown, or in a sequence. Thediscovery start packets request each host to return one or more packetsto a specified IP address and part of the directory server. The hostsreply at blocks 606 and 607, with the requested packets. The directoryserver determines at block 608, in a manner similar to that discussedabove in connection with the keep-alive process, the current state ofany firewalls. The IP address and port combinations are computed by thedirectory server on the basis of the secondary communication channelpresently established for the called party, on the basis of heuristicsor stored information defining the behaviors of known firewall systems.The directory server then sends at blocks 609 and 610, a list ofcommunication parameters upon which each host will base its conferencestart packets. The parameters include IP address and port combinations,corresponding to the host, together with corresponding conference IDnumbers. The IP address and port combinations include the expectedvalues for one or more types of firewalls, preferably including theexpected values for all known types of firewalls, given the stat of thesecondary communication channel through the actual firewall upon whichthe calculation is based.

At blocks 611 and 612, each host then sends a conference start packet toeach of the combination of IP address and port for the conference IDgiven in the list just received at blocks 609 and 610.

It is expected that at least one conference start packet will getthrough in each direction. When each host receives a conference startpacket directly from the other, at block 613, a two-way directcommunication between the calling party and called party is establishedthrough a primary channel. No keep-alive process is required to keep theprimary channel open, even if packets become lost en route from one hostto the other, because each host is continuously sending data to theother, thus keeping their own firewall open for return packets from theother.

The keep-alive process between each host and the directory server,described above, however, continues in the background, to insure thatthe secondary channel is not lost in the event that the number andperiod of the packets transmitted between each host and the directoryserver falls below the threshold of one of the firewalls for keeping aport open. This allows the directory server to initiate communicationswith the hosts, for example to update the list of contacts available tothe host when a contact registers or deregisters with the directoryserver.

If the process described fails to produce a successful direct connectionfor a conference, then any suitable retry, recovery or fall-back methodmay be employed to improve the chances of successfully establishing thedirect connection. In the case of such a failure, the key elements ofcommunication between each of the hosts and the directory server areused in the retry, recovery or fall-back method, as it is those elementsthat permit direct communication of unexpected messages, such as theconference start message, to a host protected by a firewall.

Having thus described several aspects of at least one embodiment of thisinvention, it is to be appreciated various alterations, modifications,and improvements will readily occur to those skilled in the art. Suchalterations, modifications, and improvements are intended to be part ofthis disclosure, and are intended to be within the spirit and scope ofthe invention. Accordingly, the foregoing description and drawings areby way of example only.

1. A method for communicating video-conference communication databetween a first host and a second host, comprising: at the first host,encoding at least a portion of the video-conference communication dataproducing a first encoded video-conference signal; transmitting, by thefirst host, the first encoded video-conference signal to the second hostfor processing and display at the second host, wherein encoding andtransmitting are both performed by a first single multimedia thread ofexecution of an operating system at the first host; at the first host,receiving a second encoded video-conference signal from the second host;and decoding, by the first host, the second encoded video-conferencesignal producing a decoded video-conference signal, wherein decoding andreceiving are both performed by a second single thread of executionseparate from the first single multimedia thread of execution, at thefirst host, wherein the first single multimedia thread issingle-threaded to only perform encoding and transmission functionalityof the first host, and wherein the first host is dual threaded such thatlatency in the first host is minimized.
 2. The method according to claim1, wherein the method comprises performing a callback, by the firstsingle multimedia thread, to perform encoding and transmitting.
 3. Themethod according to claim 1, wherein the second single thread ofexecution is a second single multimedia thread of an operating system ofthe first host, and wherein the method comprises performing a callback,by the second single multimedia thread, to perform decoding andreceiving.
 4. The method according to claim 1, further comprising:sending an audio signal to the second host independently fromtransmitting the first encoded video-conference signal to the secondhost, the audio signal comprising at least a second portion of thevideo-conference communication data.
 5. A system for communicatingvideo-conference communication data over a data path between a firsthost and a second host comprising: in the first host, a processorexecuting a software program performing the method of claim
 1. 6. Thesystem according to claim 5, wherein the method comprises performing acallback, by the first single multimedia thread, to perform encoding andtransmitting.
 7. The system according to claim 5, wherein the secondsingle thread of execution is a second single multimedia thread of anoperating system of the first host, and wherein the method comprisesperforming a callback, by the second single multimedia thread, toperform decoding and receiving.
 8. The system according to claim 5,further comprising: an audio signal path connecting the first host andthe second host, independent of the data path.
 9. A computer-readablemedium having stored thereon a plurality of machine instructions forcontrolling at least one system to perform a method, the methodcomprising: at a first host, encoding at least a portion of thevideo-conference communication data producing a first encodedvideo-conference signal; transmitting, by the first host, the firstencoded video-conference signal to a second host for processing anddisplay at the second host, wherein encoding and transmitting are bothperformed by a first single multimedia thread of execution of anoperating system at the first host; at the first host, receiving asecond encoded video-conference signal from the second host; anddecoding, by the first host, the second encoded video-conference signalproducing a decoded video-conference signal, wherein decoding andreceiving are both performed by a second single thread of executionseparate from the first single multimedia thread of execution, at thefirst host, wherein the first single multimedia thread issingle-threaded to only perform encoding and transmission functionalityof the first host, and wherein the first host is dual threaded such thatlatency in the first host is minimized.
 10. The computer-readable mediumof claim 9, wherein the method comprises performing a callback, by thefirst single multimedia thread, to perform encoding and transmitting.11. The computer-readable medium of claim 9, the method furthercomprising: wherein the second single thread of execution is a secondsingle multimedia thread of an operating system of the first host, andwherein the method comprises performing a callback, by the second singlemultimedia thread, to perform decoding and receiving.
 12. Thecomputer-readable medium of claim 9, the method further comprising:sending an audio signal to the second host independently fromtransmitting the first encoded video-conference signal to the secondhost, the audio signal comprising at least a second portion of thevideo-conference communication data.